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ABSTRACT 

A review of cost*ef fcctiveness analysis techniques 
was conducte'l in order to facilitate the selection of the most 
appropriate tool to aid any given instructional development effort. A 
survey vas made of the cost-effectiveness techniques in current use, 
the differences among these techniques were examined in an analytical 
framework, and criteria for the selection of appropriate techniques 
were developed« The study yielded the conclusion that the variety of 
methods is due to choices available regarding 1) the level and 
process of analysis and 2) the analytical methods which can be 
employed. Selection of level and process was found to be generally 
beyond the control of the analyst, usually being deterninod by the 
nature of the study or the requirements of the requestor. However, a 
certain latitude in cost-ef fectivenesss analytical method selection 
was felt available to the analyst. This latitude was determined by 
the characteristics of the analytical techniques available for use 
and these, in turn, were identified as the tabular display technique 
and the analytical model technique. (Author/LB) 
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FORWARD 



Cost-effectiveness analysis is becoming increasingly 
important as a decision-assisting tool to both government and 
industry. Cost-effectiveness analysis also has been a signi- 
ficant factor in the selection of alternative design in instruc- 
tional development. The purpose of this monograph is to pro- 
vide an overview of what is meant by cost-effectiveness analy- 
sis and to define the essential elements of the methodology for 
trade-off decisions by the instructional designer. The mono- 
graph includes a documentation and synthesis of the work of a 
variety of authorities, and presents their collective exper- 
iences ► The treatment of this documentation and synthesis is 
to structure the information into a narrative and graphic des- 
cription of the analytical process of cost effectiveness. Al- 
though the description includes considerable detailed informa- 
tion involving the documentation of steps in the various analy- 
tical techniques, the description is still less than would be 
required to perform all aspects of the steps by an instructional 
designer. Tne degree of detail maintained in this description, 
therefore, is considered to reflect the minimum information re- 
quired for the performance of cost-effectiveness analysis by an 
instructional designer with moderate experience. 
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PREFACE 



The purpose of this monograph is to identify the differ- 
ences in cost-effectiveness analysis methods so that the most^ 
appropriate method can be selected as a decision-assisting tool 
in instructional design and development. It is intended that 
this approach will be of assistance to those instructional de- 
signers engaged in selecting alternative instructional strate- 
gies by providing the following: 

• Insight into the field experience of others in 
instructional cost-effectiveness analysis. 

• Recommendations for direct application of cost- 
effectiveness analysis principles to instruc- 
tional design. 

• Source information on instructional cost-effec- 
tiveness analysis, as well as reference to in- 
structional designers with experience using cost- 
effectiveness analysis. 

The principal organization of the monograph Is to explore 
and identify what cost-effectiveness methods were being used; 
to identify and examine the differences among these methods in 
an analytical framework; and, to determine the criteria to be 
used in selecting the most appropriate instructional cost-effec- 
tiveness method for instructional design and development. The 
emphasis of the monograph is on the performance of instructional 
cost-effectiveness analysis as a service in support of the de- 
cision-making process by someone other than the final decision- 
maker. The descriptive detail provided in^ the study is such 
that an instructional designer with moderate experience could 
perform the indicated activities. 
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ABSTRACT 



Much has been reported on the advantages of cost-effec- 
tiveness analysis as a decision-assisting tool for instructional 
development, but little has been reported on the actual perfor- 
mance of cost-effectiveness analysis itself. The study is struc- 
tured to provide answers to the following questions: 

• What cost-effectiveness analysis methods are 
available for use and how do they operate? 

• What are the characteristics of the various 
cost-effectiveness analysis methods and in 
what way are the various methods different? 

• What factors are to be considered in the selec- 
tion of a cost-effectiveness analysis method 
for a specific situation? 

What appears to be a large variety of cost-effectiveness 
analysis methods is due to a choice in the "level" of analysis; 
in the process of analysis; and, in the analytical methods em- 
ployed within what might be called a standard step procedure. 
The choice of "level" and process of analysis is usually out- 
side of the control of the cost-effectiveness analyst. This is 
because both /'level" and process of analysis were found to be 
determined by the nature of the study or by the requirements of 
the requestor. Certain latitude in cost-effectiveness analyti- 
cal method selection is available to t:he analyst and this lati- 
tude is determined by characteristics of the analytical tech- 
niques available for use. These techniques are the tabular 
display technique and the analytical model technique. 
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INTRODUCTION 



Education currently consumes a vast amount of resources, 
and it appears almost certain that it will consume much more in 
the future. Since the systems analysis process makes possible 
the evaluation of alternative courses of action for the achieve- 
ment of instructional goals and objectives, those involved in 
instructional design and development are enthusiastic over its 
potential as a viable base for decision-making. Cost-effective- 
ness analysis, one aspect of systems analysis, has been shown 
to aid in the better use of resources. 



NEED FOR STUDY 



It would seem that the current position on system/cost- 
effectiveness analysis is that many are enthused over its po- 
tential (McGivney, 1969, p. 31)--but who will perform the analy- 
sis and how the analysis will be done are more difficult consi- 
derations. It is possible that systems/cost-effectiveness per- 
sonnel could be employed from the following areas: 



t Government. Those individuals associated with 
the Department of Defense and their prime con- 
tractors who hope to transfer their professional 
and scientific capabilities from defense to edu- 
cation-oriented activities (Lieberman, 1968, p. 24). 

• Business. Independent consultants with no vested 
interest in hardware or a product (Herbert, 1967, 
p. 45) . 



• Education. Educators and administrators res- 
ponsible for planning (Heinich, 1969, p. 330) 
and instructional designers (Allen, 1967, p. 
377). 

Attaining a process by which cost-effectiveness analysis 
may be applied to instructional design and development, however, 
is a more formidable task. One difficulty is that although much 
has been reported on the advantages of instructional cost-effec- 
tiveness, only a limited number of "how to do cost-effectiveness" 
studies have been published (Fisher, 1967, p. 69). Another pro- 
blem is that the usefulness of cost-effectiveness analysis is 
not fully understood by many instructional designers. This lack 
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of understanding is due to the apparent coldness of the tool 
as a measure of decision-makinp, (Banghart, 1969, p. 206) as 
well as other reasons. The following observations document 
these other reasons : 

• At the current state of development of analy- 
tical methods , cost -effectiveness analysis is 
an art rather than a science. Long range plan- 
ning decisions are too complex for the current 
state of analytical art to handle, 

• Cost-effectiveness analysis is difficult to 
understand because of the confusion of seman- 
tics and meaning (Fisher , 1967 , pp. 66, 68 

§ 70) . 

• The reliability of cost-effectiveness analysis 
is closely correlated with the experience of 
the individual analyst. Lack of consistency 
between analytical techniques results in ques- 
tioning the merits of cost-effectiveness as an 
aid to decision-making (Kazanowski, 1968, p. 
113). 

• Cost-effectiveness analysis takes up too much 
time to be useful (Fisher, 1967, p. 67) and is 
not worth the bother of using it. Cost-effec- 
tiveness analysis itself should be the subject 
of a cost-effectiveness study (Hartley, 1968, 
pp. 235-236) . 

There are a variety of cost-effectiveness analysis methods in 
use today which differ in their respective advantages and limi- 
tations (Seller, 1969, pp. 1-2). As new approaches develop, 
these methods constantly undergo subtle, but significant, changes 
(Kazanowski, 1968, p. 163). Although most of these new approaches 
are primarily mathematical in nature, a fev; are directed toward 
a better treatment of problems that cannot be handled by purely 
quantitative methods (Quade, 1968, p. 242). 

With the apparent lack of understanding, limitations, 
criticism, and wide differences of approaches in cost-effective- 
ness analysis, the following question seems appropriate. How 
can this analytical tool, which was designed as a decision-assist- 
ing aid in the selection of technical systems, be used in efforts 
to attain the less tangible objective of increasing student 
achievement (Freedman, 1969, p. 31)? 
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PURPOSE OF STUDY 

The principal purposes and organization of this mono- 
graph are to: (1) explore and identify what cost effective- 
ness methods are being used, (2) identify and examine the 
differences ox these methods in an analytical framework which 
accounts for the major variables identified, and (3) deter- 
mine the criteria to be used in selecting the most appropriate 
cost-effectiveness analysis method for instructional design and 
development. As noted, there are a variety of cost-effective- 
ness analysis methods and these differ in approach and precision 

It is suggested that differences in co3t-ef f ec tiveness 
analysis methods can be identified, so that the most appropriate 
analytical method may be selected as a decision-assisting tool i 
instructional design and development. The monograph will answer 
the following questions: 

• What cost-effectiveness analysis methods are 
available for use and how do they operate? 

• What are the characteristics of the various 
cost-effectiveness analysis methods and in what 
way are the various methods different? 

a What factors are to be considered in the selec- 
tion of a cost-effectiveness analysis method 
for a specific situation? 



LIMITATIONS 

Due to the complexity of the area of systems/cost analy- 
sis, the study will be limited to the process of cost-effective- 
ness analysis itself. In addition, the emphasis v;ill be on the 
performance of the cost-effectiveness analy*::is by someone other 
than the final decision-maker, as an instructional design and 
development service in support of instructional management. 
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COST-EFFECTIVENESS ANALYSIS METHODS 



The purpose of instructional cost-effectiveness analysis 
is to provide the decision-maker with data on the cost (resource 
consumption) and the probable effectiveness (quality of results 
to be achieved) for each of several alternatives, among which 
he must choose one course of action. Cost-effectiveness analy- 
sis involves clarifying the relationships between these two 
factors (cost and effectiveness) so that the decision-maker can 
strike the best balance between them. Cost-effectiveness analy- 
sis is not a substitute for judgment, experience, and common 
sense. It is a method for the systematic examination of all 
the subjective and objective data available (Heymont, 1965, 
p. 1). 



PERFORMANCE 

The current trend in the performance of cost-^fCective- 
ness is the application of the method by all levels of decision- 
making, in a variety of functions, and for both short-range and 
long-range planning (Ileuston § Ogawa, 1966, p. 243). The analy- 
sis itself may be performed by the decision-maker, by a special- 
ist within the decision-maker * s organization, or the responsi- 
bility for cost-effectiveness analysis may be delegated to a 
specialized service organization (McGivney, 1969, p. 31). The 
latter approaches require close cooperation and coordination 
between the cost-effectiveness analyst primarily responsible 
for the study, tho decision-maker requesting the study, and 
each of the other individuals providing informational inputs 
(Fields, 1966, p. 517) . 

At best, cost-effectiveness analysis functions as a tool 
used in providing the decision-maker with an analytical founda- 
tion for making sound objective choices among the various ways 
a problem might be solved or an objective met (Jarrett, 1967, 
pp. 7 a 9). 



DEFINITION 

The definition of cost-effectiveness analysis varies, 
for the methodology is not precise enough to mean the same thing 
to all practitioners in the field (Seller, 1969, p. 1). Broadly 
defined, it is an analytic study designed to assist a decision- 
maker in identifying a preferred choice among possible alternates 

O 
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(Quade, 1969, p. 1). The choice of alternates involves two 
approaches: (1) fixed-cost and flexible effectiveness ap- 
proach; an ' {2} fixed-effectiveness and flexible cost ap- 
proach (K. ])Ws]ci, 1968, pp. 126-127). 

Attention should also be given as to what is not cost- 
effectiveness analysis. Kazanowski calls this the "max'imum- 
ef f ectiveness-at-minimum-cost fallacy" and defines this mis- 
conception as follows: 

Many references can be found in which the state- 
ment is made directly or indirectly that cost-effective- 
ness is a technique by which one determines that alter- 
native which yields the maximiun effectiveness at a mini- 
mum cost. Or, in other words, that alternative is sought 
by which the cost is minimized. In reality the attempt 
to find such an alternative is doomed to failure at the 
onset, because such an alternative does not exist. Hitch 
and McKean state: "Actually, of course, it is impossible 
to choose that policy which simultaneously maximizes 
gain and minimizes cost, because there is no such policy" 
(Kazanowski, 1968, p. 160). 



EFFECTIVEMESS/BENEFIT DIFFERENCES 

Confusion also exists concerning the difference between 
cost-effectiveness analysis and cost-benefit analysis (Fisher, 
1967, p. 66). The prinicpal differences in these two methods 
are the elements of output and of time-phasing. These differ- 
ences are diagramed in Figures 1 and 2. 



Output 

The term "cost-effectiveness" usually applies to situa- 
tions in which it is difficult to quantify the alternate out- 
puts in terms of dollar values. Cost-benefit usually consi- 
ders only the monetary worth of outcomes. 



Time Phasing 

The term "cost-effectiveness" is usually used to select 
the most appropriate choice from the predicted alternative out- 
comes; the term "cost-benefit" is most often applied in select- 
ing the most appropriate choice from the calculated present 
(McGivney and Nelson, 1969, p. 105). 
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SUBJECT APPROPRIATENESS 

Certain prerequisites should be met before a subject is 
appropriate for cost-effectiveness analysis (Kazanowski, 1968, 
p. 114). The primary appropriate prerequisite is that the sub- 
ject should have two character isitcs : first, and most impor- 
tant, the subject must be relevant; second, but less important, 
it should be capable of being measured both in required re- 
sources and outcome. These two characteristics often appear 
conflicting. The most relevant phenomena or events are often 
very difficult to measure, the least relevant are usually easy 
to measure (Niskanen, 1969, p. 20). Cost-effectiveness analy- 
sis should be performed only when the following appropriate 
prerequisites are met: 

• The correct problem must be recognized. 

• Realistic objectives that bear a functional re- 
lationship to the problem must be determined 
(Jakobsberg, 1966, p. 38). 

• Common goals, objectives, or missions of the al- 
ternative must be identifiable and at least 
theoretically attainable. 

• Constraints for bounding the problem must be 
discernible . 

• Alternative means of meeting the goals must 
exist (Kazanowski, 1968, p. 114) . 

The meeting of these requirements is not always simple; 
consequently, if there is no freedom of choice in problem solv- 
ing, cost-effectiveness analysis is not appropriate (Jarrett, 
1967, p. 8). 



RELATIONSHIP TO SYSTEMS ANALYSIS 

la order to place cost-effectiveness in proper perspec- 
tive, it is first necessary to place it in relation to the sys- 
tems analysis process. Cost-effectiveness analysis is not a 
method for measuring either cost or effectiveness after the fact, 
for its modern applications are concerned with the prediction 
and evaluation of an alternative's worth (English, 1968, p. 7). 
The term "systems analysis'' is used to distinguish this broader 
analysis from a narrow cost-effectiveness aspect of analysis. 
Cost-effectiveness analysis is^ that aspect of systems analysis 
which emphasizes the systematic investigation of the decision- 
maker's alternatives relative to objective criteria, costs, and 
effectiveness comparison, in addition to the other trade-off 
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factors associated with alternative selection (Quade, 1969, 
p. 3). Figure 3 graphically illustrates this emphasis. 

It can be seen, then, that the difference between 
cost-effectiveness analysis and systems analysis is a ^ 
matter of emphasis. If the emphasis is on finding sig- 
nificant differences in the costs or resource require- 
ments among the available alternatives for carrying out 
some specified task, the analysis is gnerally referred 
to as a cost-effectiveness analysis. The systems analyst, 
on the other hand, is likely to be forced to deal with 
problems in which the difficulty is deciding what ought 
to be done, not simply how to do it. System analysis 
thus puts greater emphasis on the suitability of the 
task and the augmentation of alternatives. In both sys- 
tem-analysis and cost-effectiveness work it is important 
to note, however, that after an appropriate evaluation 
one will be in a much better position to make a decision, 
even though he may not find the decision easier to make 
(Bell, 1964, pp. 1-2) . 



CATEGORIES 

The confusion over the role cost-effectiveness analysis 
plays in the systems analysis process seems to arise from the 
fact that some practitioners perceive cost-effectiveness 'analy- 
sis to be the systems analysis process itself; whereas, others 
see the process of cost-effectiveness as being related to and 
a sub-sct of the total systems design process. As noted above, 
both interpretations of the role of cost-effectiveness analysis 
could be appropriate, depending upon the time and conditions 
that the analysis requires • There are, however, three categor- 
ies of cost-effectiveness analysis which, in practice, are not 
clearly distinct. These categories are: (1) system/compo* 
nent configuration study; (2) system comparison st?idy; and, 
(3) suprasystem comparison study (Heustcn ^ Ogawa, 1966, p. 245). 
The relationship of these categories is exhibited in Figure 4 
and a detailed description follows. 



Systems Configuration/Component Study 

In this category the emphasis is on the selection of the 
particular configuration, or characteristics, of a single sys- 
tem with different components. For example, r closed-c ircuit 
instructional television system versus an open-circuit instruc- 
tional television system. This category of study is probably 
the one most often performed. 
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Cost Treatment. Typically, system/component configura- 
ticn studies have the following characteris ics relating to the 
treatment of costs: 

• Detailed costs are obtained on all components. 

• Various mathematical models are used to simu- 
late and optimize different possible configura- 
tions of the components. The results yield the 
predicted overall costs of the various alterna- 
tives under study. 

• Costs are compiled by summing the yearly costs 
over the life cycle of the systems rather than 
by compiling time-phased costs. 

Effectiveness Treatment. As in the treatment of cost, 
the effectiveness measure used in the system configuration/ 
component study will be detailed in terms of capabilities and 
performance. In most cases, effectiveness will not be a sim- 
ple overall measure which ultimately would determine alterna- 
tive selection (Hatry, 1969, pp. 45, 47-48). 

It should be noted that trade-off analysis at the detail 
level of selecting subcomponents is classified as value analy- 
sis rather tlian cost-effectiveness analysis (Kazanowski, 1968, 
p. 153). An example of this type of analysis would be the selec- 
tion of the least expensive instructional television receiver 
that complies with given specifications. 

System Comparison Study 

Here, the emphasis is on comparing two or more systems 
for the same objective. The emphasis is on mter system, not 
intrasystem, analysis. It is presumed that each competing sys- 
tem has already been sub-optimized as to its configuration, 
through the system/configuration component study discussed pre- 
viously. An example of this type of study would be to compare 
the merits of a programmed-instructional system versus an auto- 
tutorial system (where each has been instructionally validated) • 



Cost Treatment. Typically, the system comparison study 
has the following characteristics relating to system costs: 

• Costs are generally required in less detail than 
in system/component configuration studies where 
the emphasis was on components of competing 
similar characteristics. 
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• The specific spread of 
cycle of the system is 
treated as a secondary 



costs over the life 
usually ignored, or is 
problem. 



Effectiveness Treatment. The effectiveness measure is 
basically quite similar to that used in system/component con- 
figuration studies. 



Suprasystem Comparison Study 

This third cost-effectiveness study category is less 
frequently performed as a strictly cost-effectiveness activity. 
Activities at this level of analysis are usually described as 
cost-benefit. The problem here is to assess the merits of al- 
ternatives of widely differing capabilities and cost elements. 
An example would be to analyze the cost^ef f ectiveness of com- 
peting instructional systems such as institutional study as 
opposed to on-the-job training. 



Cost Treatment. In suprasystem comparison studies, 
costs are each time-phased to the extent possible, to provide 
estimates of annual resource requirements (Hatry, 1969, pp. 48 
§ 50). 



Effectiveness Treatment. The determination of a measure- 



ment of effectiveness is especially difficult for those cases 
where the systems are experimental, the objectives are of un- 
equal priority, and the output varies in type and reliability 
(Heuston 5 Ogawa, 1966, p. 245). 

The problem of the analysis of differing costs and cap- 
abilities of alternatives requires the development of some mea 
sure of structure for comparing these alternatives on a some- 
what equal basis (Department of Defense, 1971, p. 5). 



PROCESSES 

In addition to the confusion from the differences of 
cost-effectiveness analysis categories, differences in the pro- 
cess of the analysis itself has compounded the lack of under- 
standing. Yet, in all the variations, two processes are per- 
formed: (1) the procedural process; and, (2) the application 
process . 
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Procedural Process 

This cost-effectiveness analysis process can be defined 
as the documentation or writing of instructions on how to per- 
form the analysis in a step procedural sense. This type of 
process is also known as model building; i.e., a conceptual 
mock-up or detailed paper-pencil description of the actual 
analysis to be performed at some later time. 



Appl ication Process i 

The second cost-effectiveness analysis piocess refers 
to the application of the procedural process by applying a 
particular model to a real -world practical situation fSpeagle, 
1969, p. 22) . 



STANDARD PROCEDURE 

The chief value of cost-effectiveness analysis is its 
broad ability to provide a general framework to assist in the 
solution of problems in a wide range of topical areas. Cost- 
effectiveness analysis differs from many other decision-assist- 
ing tools which are limited to accepting only certain classes 
of problems. 

The very fact that cost-effectiveness analysis can as- 
sume characteristics dictated by a unique objective and respond 
to the category, level, and process of analysis as required 
provides a lack of uniformity in the analysis itself (Jakobs- 
berg, 1966, p. 37). In addition, the apparent lack of uniform- 
ity in most examples of cost-effectiveness analysis, and the 
lack of consistent documentation, has led some persons to ques- 
tion the merits of this type of analysis as an aid to decision- 
making (Kazanowski, 1968, p. 113). 

On the other hand, certain authors and authorities in 
the field indicate that a complete "how- to-do- it" document, or 
a "cookbook," would be impossible to prepare (Jakobsbcrg, 1966, 
p* 37). This may bo because, at the current stage of develop- 
ment, cost-effectiveness analysis is more an art than a science 
(Fisher, 1967, p. 70). Nevertheless, it is suggested that the 
majority of approaches to cost^-ef fectiveness analysis have cer- 
tain common functional elements which may be synthesized into 
a standard procedural approach. These steps are illustrated in 
Figure 5 and include the following inputs from others, analysis, 
and outputs: 

• Step 1. Review definition of the desired 
general goals and specific objectives that the 
design is to meet or fulfill* 
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STEP 1 

REVIEW DEFINITION 
OF GENERAL GOALS/ 
SPECIFIC OBJECTIVES 



STEP 2 

REVIEW IDENTIFICA- 
TION OF PARAMETERS 
OR REQUIREMENTS 




STEP 3 

REVIEW DEVELOPMENT 
OF ALTERNATIVES 




STEP 4 

ESTABLISH COST 
AND EFFECTIVENESS 
EVALUATION 
CRITERIA 



STEP 5 

DETERMINE COST 
OF ALTERNATIVES 



I 



STEP 6 

DETERMINE CAPA- 
BILITIES OR 
EFFECTIVf:NESS OF 
ALTERNATIVES 



STEP 7 

SELECT FIXED- 
COST OR FIXED- 
EFFECTIVENESS 
APPROACH 





STEP 8 




CREATE 


ALTERNATIVE 


VERSUS 


CRITERIAN 


ARRAr 





STEP 9 

analyzl 'merits 
of alternatives 



I 



STEP 10 

ANALYZE UNCER- 
TAINTY FACTORS 



STEP n 

DOCUMENT BASIS OF 
PREVIOUS STEPS; 
SUBMIT FINDINGS 



OUTPUT 



Note: After Pearson, 
1972, p. 35 



Figure 5. Standard Procedure for Analysis 
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Step 2. Review identification of the para- 
meters or requirements essential for the attain- 
ment of the desired goals/objectives. 



• Step 3. Review development of the alterna- 
tives for accomplishing the goals/objectives/ 
requirements . 

• Step 4. Establish an evaluation criteria (mea- 
sures) , for cost and effectiveness, that relate 
alternative capabilities to the goals/objectives/ 
requirements . 

• Step 5. Determine the cost of alternatives 
in terms of evaluation criteria. 

• Step 6. Determine the capabilities or effec- 
tiveness of the alternatives in terms of eval- 
uation criteria. 

• Step 7. Select fixed-cost or fixed-effective- 
ness approach. 

• Step 8. Create an alternatives versus criter- 
ian array. 

• Step 9. Analyze the merits of alternatives. 

• Step 10. Analyze the uncertainty factors. 

• Step 11. Document the rationale, assumptions, 
and analyses underlying the previous steps and 
submit findings (Pearson, 1973, p. 34) . 

The following discussion of these procedures suggests guidelines 
to be followed for a standardized approach to the analysis of 
system/cost-effectiveness. Although the presented steps are 
in an order in which they would usually be performed, changes 
in the sequence are acceptable and should fit the need of the 
subject under evaluation (Kazanowski, 1968, pp, 115-116). 

Step 1 

Review definition of the goals/objectives. Before cost- 
effectiveness analysis can be considered, the problem must be 
identified and the goals/objectives must be defined, at least 
in a general nature. Obvious as this may seem, the perception 
of the problem that needs solving, and the establishment of 
goals and objectives, are very often complex and involved tasks 
for which there are no techniques and tools besides "logical 
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thinking" (Jakobsberg, 1966, pp. 37-38). Without such an iden- 
tification of goals/objectives there is no framework for struc- 
turing the subsequent evaluations. The following are several 
points to consider in reviewing the general goals/objectives 
definition. 

goals/objectives specified are in too general terms, 
the constraints established for bounding the evaluation are 
often only the product of the analyst. In addition, care must 
be exercised not to identify the goals/objectives in such a 
manner as to bias the evaluation by including requirements of 
such specific nature that they exclude potential candidate 
alternatives from consideration. On the other hand, care must 
also be taken not to make goals/objectives too specific or they 
will limit the scope of possible candidate alternatives by im- 
plicitly defining alternative concepts rather than just the de- 
sired goals/objectives, A potential danger always exists in 
that the goals/objectives originator may specify a goal/objec- 
tive that is unattainable by means of current technology. 



Step 2 

Review the identification of the parameters or require- 
ments. The basic purpose of identifying and defining the gen- 
eral goals/objectives is to aid in the identification of re- 
quirements essential to attaining the defined goals/objectives. 
The goals/objectives and requirements should be identified and 
specified within the required parameters. This specification 
is used to further reduce the possibility of biasing the cost- 
effectiveness analysis. The confusion brought about by close 
relationship between goals/objectives and requirements is fur- 
ther compounded by the variety of categories of cost-effective- 
ness studies (i.e., suprasystem comparison study and system 
configuration/component study) . The need for detailed speci- 
ficity can be illustrated by the following simplified example 
of the general goal/objective of eliminating poverty of migrant 
agricultural workers by vocational skill training. One require- 
ment would be the identification of current labor market voca- 
tions needed so that detail job performance training specifica- 
tions could be formulated. The resulting requirement specifi-- 
cation might detail tool or machine operator competencies need- 
ed, as well as conditions and level of performance expected 
after training completion. The major factors for consideration 
when reviewing the identification of goals/objectives and re- 
quirements, and the development of specifications are as follows 

• Verify the es tablisliment of a relationship that 
converts goals/objectives into some unit of mea- 
sure for both cost and effectiveness. For exam- 
ple, number completions at a given level of 
achievement. 



• Be alert for errors of commission, which are 
just as important as errors of omission. If 
goals/objectives that are not necessarily es- 
sential are identified as requirements, they 
can strongly prejudice the subsequent cost- 
effectiveness evaluation (Kazanowski, 1968, 
pp. 117-119) . 



Step 3 

Review development of the alternatives. As in defining 
the goals/objectives and identifying the requirements, develop- 
ing valid alternative means or courses of action that can sat- 
isfy the goals/objectives is a mental exercise for which no 
current scientific technique exists (Jakobsberg, 1966, p. 39). 
This activity is to create two or more alternative concepts of 
ways to achieve the goals/objectives within the defined para- 
meters or specifications. Frequently, considerable imagination 
and creativity are required to develop effective competitive 
candidate alternatives. The results of cost-effectiveness an- 
alysis can be no better than the conception and development of 
attractive candidate alternatives (Kazanowski, 1968, p* 120). 

There arc several problems associated with the review of 
the alternative development. The most serious problem or source 
of defects is attention bias. This is frequently caused by a 
cherished belief or an unconscious adherence to what we might 
call a "party line." This attention bias may result in the un- 
warranted favoring of a particular alternative by the analyst 
(Quade, 1969, p. 8). Another problem is the degree of detail 
associated with available information on alternatives. Since 
the purpose of cost-effectiveness analysis is to aid the deci- 
sion-maker ii> deciding which alternative should be selected, 
specific details on possible alternatives are generally lacking.^ 
Too little detail usually results in a large variance in esti-v " 
mated alternative effectiveness and cost. On the other hand, 
to require that the candidate alternative be designed in detail, 
before being evaluated, would defeat the basic purpose and value 
of cost-effectiveness analysis. In addition, the development 
of new and novel alternatives might be discouraged by detail de- 
sign requirements. This factor would tend to favor existing or 
more traditional alternatives. The balance between inadequate 
detail and excessively rigorous specificity should be maintained 



Step 4 

Establish cost and effectiveness evaluation criteria. 
The criteria specifies: (1) the measure of cost; (2) the 
measure of effectiveness; and, (3) the relationship between 
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them. All will be used in the selection of one alternative from 
the several alternatives. The measure of cost must be consis- 
tent with the overall framework of the resource allocation pro- 
blem; e.g., net future total cost of. ownership for a certain 
number of years. The measure of ef i ^^ ctiveness should express 
the extent to which the goals/objectives and requirements are 
being accomplished. The relationship between the cost and effec 
tiveness must be expressed in functional terms; e.g., in a mathe 
matic equation, graphic display, or model (Bryk, 1965, p. 1). 

The selection of appropriate and adequate cost and^effec- 
tiveness evaluation criteria is based on judgment and experience 
The omission of significant criteria could readily invalidate 
the results of a cost-effectiveness analysis. The inclusion of 
numerous criteria to be on the safe side, however, can result 
in mental paralysis when the optim\im alternative selection is 
being made. The greater the number of criteria, the greater 
the probability that even unlikely candidates will excel in 
some criterion, making alternative system selection difficult. 
One simple test of the adequacy or completeness of cost and 
effectiveness evaluation criteria is to question whether one 
alternative could excel in most of the criteria established and 
still not be selected as the preferred choice. If the answer 
is affirmative, important criteria could be missing. Consider- 
able insight into the subtleties of the goals/objectives and 
requirements is usually necessary for the establishment of 
meaningful alternative evaluation criteria (Kazanowski, 1968, 
p. 123) . Certain evaluation criteria of cost and effectiveness 
suggested by a variety of authors are shown in Figure 6. 



Substep 4.1 

/ 

Develop measurers of cost. The choice of a particular 
alternative for accomplishing^the goals/objectives implies that 
certain specific resources will^be consumed and could no longer 
be used for other purposes. These are the costs. In future 
time period analysis most costs could be indicated as money ex- 
penditures, but their true measure should be expressed in terms 
of the values of alternative opportunities which are precluded 
because of this expenditure (Quade, 1969, p. 4). One of the 
most difficult problems in establishing cost evaluation criteria 
is the determination of appropriate cost measures. The choice 
of the proper cost measure involves both difficulties and con- 
troversy within the broad categories of time-phased analysis 
(cost distributions over extended time periods) and non-time- 
phased analysis. 



Cost Measures for Non-Time-Phased Analysis. For the pur- 
pose of non-time-phased analysis (such as applied to the system/ 
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MEASURES OF COST 



I. STATIC 
li. TIME-PHASED 



] 



A. SYSTEM COST TO ACCOMPLISH 
GOALS/OBJECTIVES 

B. FUNDING RATE 

C. RESOURCES REQUIRED 

D. DISCOUNTING 



1. RESEARCH AND DEVELOPMENT 

2. INITIAL (MFG., DEPLOYMENT, TRAINING) 

3. OPERATING (ADMINISTRATIVE, LOGISTICAL) 

4. OTHER 



MEASURES OF EFFECTIVENESS 



I. STUDENT/GROUP-ORIENTED| 
II. SCHOOL-ORIENTED 
III. COMMUNITY ORIENTED 



A. UTILITY 

B. PRODUCTIVITY 

C. WORTH 

D. MERIT 

E. BENEFIT 

F. GAIN 

G. VALUE RECEIVED 



1 . 


PERFORMANCE 


n. 


REpAlliAyiLllV 


2. 


ECONOMY 


12. 


GROWTH POTENTIAL 


3. 


SAFETY 


13. 


DEPENDABILITY 


4. 


AVAILABILITY 


14. 


CAPABILITY 


5. 


FLEXIBILITY 


15. 


TECHNICAL CONFI- 


6. 


PRESTIGE 




DENCE 


7. 


MAINTAINABILITY 


16. 


INFORMATION YIELD 


8. 


RELIABILITY 


17. 


VERSATILITY 


9. 


PROBABILITY OF 


18. 


SPILLOVER 




SUCCESS 




EFFECTS 


10 


.EVOLUTIONARY 


19. 


TECHNICAL 




DEVELOPMENT 




DESIRABILITY' 



Note: After Pearson, 1972, p. 42 



Figure 6. Cost and Effectiveness Evaluation Criteria 
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component level co3if iguration or system level comparison study) 
the folloMing throe cost measures are commonly suggested. 

• Initial cost (inclutfllnp, research and develop- 
ment and initial in\restment, to achieve the 
assumed goals/objectives plus X years of oper- 
ating costs. The clhoice of what X should be 
is a problem, but it is most commonly taken as 
five and occasionally ten years. 

• Adjii^t§d initial cost plus X years of operat- 
ing costs. The difference here is only the 
use of adjusted initial cost in which case the 
costs are adjusted to take into consideration 
the estimated residual values for each alter- 
native* The residual -value problem occurs most 
frequently when comparing choices between al- 
ternatives which either: (a) could be used 
for additional goals/objectives; or, (b) the 
life-cycle of alternatives differs widely. 

The latter situation is the most often cause 
for initial cost adjustment. 

• The above first or second cost measure but dis- 
counted in some manner. Discounting has been 
used in order to perform two functions: (1) 

as an adjustment for the increasing time period 
uncertainties; and, (2) to indicate economic 
impact . 



Cost Measures for Time-Phased Analysis. For the supra- 
system comparison studies, there are three general cost measures 
suggested to be used in examining each alternative. These are: 

• Annual funding/budget requirements. 

• Cumulative funding/budget requirements. 

• Present expenditure worth at a selected discount 
rate. 

The problem in all of the above measures is to determine how 
far out the study should be carried in order to consider all 
time periods pertinent to the purpose of the study. At some 
point the effect of the subsequent years would have negligible 
effect upon today •s decisions. 

The is probably no ideal answer to the question of the 
cost measure to be used, in either tirae-phased or non- time -phased 
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analysis. No single cost measure incorporates all pertinent 
elements and it may well be desirable to use more than one 
measure to give the proper cost perspective (Hatry, 1967, pp. 
64-67) . 



Substep 4.2 

Develop measurers of effectiveness. Of all the compo- 
nents or constituents of cost-effectiveness analysis the cos t - 
measure concept has found the least acceptance as a valid mea- 
surement criterion. Less difficult was the acceptance of the 
effectiveness -measure criterion (Breckner, 1967, p. 83). Pro- 
per effectiveness criteria selection, however, is one of the 
most formidable problems of cost-effectiveness analysis. If 
the goals/objectives are not trivial, then real world facets 
are generally complex and have widespread implications; conse- 
quently the effectiveness criteria may be broad and numerous. 
If the scope of the goals/objectives is reduced, usually so is 
the number of significant criteria. Even if the scope of the 
problem is significantly reduced, it is virtually impossible to 
reduce the total cost-effectiveness analysis to a single easily 
evaluated effectiveness criterion (Kazanowski, 1968, p. 152). 
On the other hand, the more narrow and fewer in number the mea- 
sure of effectiveness criteria become, the more limited will 
be the eligible alternatives to be considered. It is often de- 
sirable, therefore, to broaden a cost-effectiveness analysis 
study to include more effectiveness criteria in order to insure 
a wider range of alternatives (Breckner, 1967, p. 58). Well- 
defined and explicit measures of effectiveness are difficult 
to develop and rare. Typically, these measures are replaced 
by two devices: (i) performance identifiers; and (2) standards. 



Performance Identifiers. These effectiveness measures 
relate to the administrative control of functions where output 
is inadequately defined. Usually, performance measures are 
specified on an activity basis to give an estimate of work per- 
formed. The performance identifier tends to be a device to 
measure work performance and is designed to detect the variations 
in work performance levels. In fact, however, performances may 
have little relation to the final output. 



Standards. These effectiveness measures are much broader 
than performance identifiers. Government procurement activities 
are characterized by the use of standards, which may be of two 
types: technical performance standards, and generalized standards. 
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Considerable overlap exists > however, and many problems are 
common to both. 

• Technical performance standards exist through- 
out government and industry and are needed to 
give a vital measure of uniformity and inter- 
changeability of parts, components, assemblies, 
and processes. For example, both the govern- 
ment Bureau of Standards and the industrial 
Underwriters Laboratories develop operational 
and performance standards for electrical items. 
These standards are observed by those manufac- 
turers who wish to market products in the 
United States. 

• Generalized standards arc identified to dis- 
tinguish them from technical, and are used when 
specific output and effectiveness measures are 
difficult to determine. For example, certain 
accrediting organizations develop guidelines 
and standards for the evaluation and certifica- 
tion of public schools, academic institutions, 
and libraries. These standards are observed 
and maintained by those wishing accreditation 
(Teitz, 1968, p. 309) . 

Unique to instructional design are three hroad categoi^ies 
of instructional effectiveness measures: (1) sludent/group- 
oriented, (2) school -oriented , and (3) community-oriented. 



Student/Group Oriented Measures. These effectiveness 
measures refer to academic achievement change (grades) resulting 
from an improvement in the instructional program. 



School -Oriented Measures. These effectiveness measurers 
are characterized by three types. 

• The change in number dropping out of rchool as 
a consequence of these achievement changes in 
a given school population in terms of changes. 

• The number selecting the various available 
courses of the study. 

• The number and quality of those graduating (as 
indicated by scores on standardized tests) . 
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Community-Oriented Measures. These effectiveness mea- 
sures are characterized by two types. 

m The average expected lifetime earnings poten- 
tial of persons with different levels of edu- 
cation (economic factors) . 

• The reduced relationship between parental socio- 
economic level and individual achievement in- 
dicating increased equality of educational op- 
portunity (social change) (Abt Associates, 
Vol. I, 1967, p. 3) . 

Although not applicable in all circumstances, the following 
ground rules may be observed in selecting appropriate measures 
of effectiveness: 

• Do not force a quantification measure of effec- 
tiveness where no sound analytical basis for it 
exists . 

• Some effectiveness criteria will be quantifiable, 
but many will not be. 

• Nonquantif iable effectiveness criteria pose no 
particular difficulty in analysis when appro- 
priate . 

• When the analysis involves a mixture of quanti- 
tative and qualitative effectiveness criteria, 
the cost-effectiveness analyst must exercise 
caution in gauging the impact of the qualita- 
tive criteria. For this reason, quantitative 
criteria are usually preferred, and the result 
is interpreted in terms of the qualitative cri- 
teria (Jakobsberg, 1966, pp. 39-40). 

A final consideration is to choose a measure of effec- 
tiveness which serves as a sufficient input to the next level 
of decision-making (Niskanen, 1969, p. 30). 

A methodological consideration discussed in Step No. 4 
dealing with measures of both cost and effectiveness is that no 
single criterion measure that is appropriate for all goals/ob- 
jectives is known. Even if a single criterion measure is adop- 
ted for all goals/objectives, the present inability to quantify 
relationships suggests the use of a multidimensional scale for 
measuring results (Packer, 1968, p. 235). 
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Step 5 

Determine cost of the alternatives. Cost and effective- 
ness of alternatives must be determined in order to rationally 
allocate resources. Costs are considered easier to determine 
than effectiveness because dollars, manpower, and materials 
are homogeneous and measurable. Accurately predicting future 
costs is not simple, however, because consideration must be 
given to uncertainty of cost estimates, the unpredictability of 
inflation, and to the probable time-value of money due to dis- 
counting. Other factors that could also be included are the 
cost of opportunities lost and the effect of uncontrolled con- 
sequences of the outcomes (Packer, 1968, p. 247). 

The problem of dollar-cost estimating often assumes so 
much importance in any cost-effectiveness analysis that the use- 
fulness of the basic concept of cost (as it relates to the an- 
alysis) is lost. As a result, accounting/budget costs may take 
on an importance which may not be justified. It should be re- 
membered that costs as prediction of the future can only be es- 
timates and we always estimate future values on the basis of 
past experience. As a consequence, it is natural that records 
of past experience arc reviewed in order to project future costs. 
Since the accounting/budget costs records usually include some 
aspects of actual past costs, a critical review of these histori- 
cal costs is recommended. This review is to ensure that past 
under or over cost expenditures will not prejudice the current 
cost estimate (English, 1968, p. 77) « 

The performance of Step No. 5 should deal with cost analy- 
sis for cost-effectiveness and similar analysis. The te^'m "cost 
analysis," in one form or other, is also used for those involved 
in budgeting, accounting, and cost-control purposes. Although 
somewhat related, these methods are not considered here (HatrV, 
1969, p. 44). In cost-effectiveness analysis, the development 
of highly detailed cost breakdowns are not required (Novick, 
1967, p. 103). It should be also noted that the preparation of 
the cost estimate is usually performed by someone other than the 
cost-eff Qctiveness analyst. It is axiomatic that the estimate 
cannot be better than the statement of the goals/objectives/re- 
quirements on which the cost estimate is based. These state- 
ments, however, sometimes reflect the cost estimator's own par- 
ticular biased interests (Novick, 1963, p. 102). In the creation 
of a cost estimate in accordance with the goals/objectivcs/re- 
quiremcnts, the cost-effectiveness analyst must verify that the 
cost estimator has provided the following cost data. 



Identify Cost Categories. These categories cover the 
source of major costs (Herd, 1965, p. 81). Measures of cost 
(developed in Step No. 4) should be indicated as well as the 
following elements: 



• Matter-energy, information, and time. 

• Human effort (student time and effort, teacher 
time and effort, administrative or technician 
time and effort) • 

• Monetary costs (short run and long run capital 
and operating activities) (Miller § Rath, 1969, 
pp. 21-24), 



Identify Resource Constraints. These constraints usually 
result in the indication that highly aggregated costs are use- 
less; that costs must be broken down into relatively small units 
which will permit a variety of cost analysis approaches. 



Calculate Costs. Each category indentified should be 
considered on an appropriate time basis as well as in total. 
(Although the alternative total cost may be within the total 
resources available, the annual expenditures may not be compa- 
tible with the budget or funding pattern.) 



Identify Possible Cost Variance. A single cost estimate 
does, not indicate unknovms relative to other costs provided by 
others. Unknown factors in cost data elements should be iden- 
tified for subsequent uncertainty analysis. 



Verify Completeness. Insure compliance of cost data ele- 
ments with a checklist (Herd, 1965, p. 81), 

Essential to the validity of all cost-effectiveness analy- 
sis is the availability of appropriate, accurate, and timely cost 
estimates. It is true that a poor cost-effectiveness analysis 
can misuse even the best cost data, but even the best of analy- 
sis cannot do much with inadequate cost data (Hatry, 1969, p. 55), 



Step 6 

Determine capabilities or effectiveness of alternatives. 

Once the effectiveness evaluation criteria have been determined 
(Step No. 4) the next step is to express the capabilities of 
the alternative systems in terms of the evaluation criteria. To 
save time, this is often performed concurrently with Step No. 5. 
Quantitative expressions of capabilities are preferred when they 
are available but qualitative expressions are acceptable when; 
they are not (Kazanowski, 1968, p. 129). As in cost estimation, 
the estimates of the capabilities cf alternative systems is 
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dependent upon information and data supplied by someone other 
than the cost -eff ectiveness analyst. Althouj;h this is both an 
advantage as well as a disadvantage, confidence in the accuracy 
of alternative system capability may be lacking and may be s^ub- 
ject to the biased interests of the estimator (Quade, 1969, 
p. 10). 



Step 7 

Select fixed-cost or fixed-effectiveness approach. In 
assessing alternatives, the procedure may now take either one 
of two analytical approaches: 

• A desired level of effectiveness Tiay be specified, 
and the analyst seeks the most economical way 

to achieve it, or 

• The level of expenditure may be specified and 
the analyst explores the effectiveness offered 
by the alternatives capabilities (Breckner, 
1967, p. 43). 

The choice of specifying either the fixed-cost or the 
fixed-effectiveness analysis approach oftentimes is the option 
of the analyst. This choice is important because, in most cases, 
real-world problems cannot be adequately dealt with in such a 
simple form as optimizing either cost or effectiveness. This is 
because in most studies unique measures of cost and measures 
of effectiveness can only be obtained by directing the analysis 
on a very small aspect of the total goals/objectives and require- 
ments. The results of such as analysis is rather superficial and 
seldom reflects the real-world goals/objectives and requirements 
from which the decision-maker must select the best alternative. 
The selection of one approach ^ therefore, may more closely re- 
flect the real -world options actually available to the decision- 
maker, whereas the selection of anotner approach might reflect 
options which are not available. Another factor in the choice 
of one approach over the other may also yield economies of analy- 
sis, depending on the availability, quality, and validity of 
data to be utilized in the cost-effectiveness analysis itself. 
The choice of fixed-cost or f ixed -ef f ectiveness approach is 
least affected by the constraints imposed by the decision-maker, 
therefore, the choice of approaches is vital to the integrity 
of the analysis. 



Fixed-Cost Approach. A basic fixed-cost approach is first, 
the development of alternative systems that can compete in meet- 
ing the goals/objectives/requirements (Step No. 3) in competi- 
tion for the given resources. Second, the number of components 



29 



of each alternative that can be developed, procured, and opera- 
tionally implemented within the fixed resources, is determined. 
Finally, the alternative which satisfies the Roals/objectives/ 
requirements to the greatest extent is identified as the pre- 
ferred choice. 



Fixed-Effectiveness Approach. A basic step in the fixed- 
effectiveness approach is first, development of a measure of the 
effectiveness expressed in terms of the requirements identified 
in the prior Step No. 2. Then, alternatives are developed to 
meet the goals/objectives as in Step No. 3. The number of com- 
ponents of each alternative necessary to attain the goals/ob- 
jectives is determined and, the costs and penalties incurred by 
each alternative are estimated. Finally, the alternative that 
exhibits the least aggregate cost penalty is identified as the 
preferred choice. 

It is suggested that if inflexible constraints are im- 
posed on the resources available, the use of the fixed-cost ap- 
proach is indicated. Conversely, if inflexible constraints are 
imposed on the effectiveness required, the use of the fixed- 
effectiveness approach is indicated. In the absence of other 
guidelines, the presence of both a large number of significant 
measures of effectiveness criteria and a small number of mea- 
sures of cost criteria usually indicates that the fixed-effec- 
tiveness approach should be used (Kazanowski, 1968, pp. 126-128). 

Note that in using either the fixed-cost and/or fixed- 
effectiveness approaches to evaluate alternatives, the measure 
of criteria (either the cost criteria or the effectiveness cri- 
teria must be held constant so as to have a common basis tor 
comparison). Methods in which both the measures of cost and the 
measures of effectiveness are allowed to vary are to be avoided 
because there would then be no common basis for comparison (Bar- 
foot, 1963, p. 3) . 



Step 8 

Create alternatives versus criteria array. Ideally, it 
would be desirable to examine the cost and effectiveness evalua- 
tion criteria of the alternatives in the real world rather than 
in the simulated world of cost-effectiveness analysis. For many 
obvious reasons, such as excessive expense and/or the unavail- 
ability of the alternatives, this is not possible (Blumstein, 
1969, p. 33). Two different analytical techniques of conduct- 
ing cost-effectiveness evaluations within either the fixed-cost 
or fixed-effectiveness approach are of ten- encountered . These 
are: (1) the tabular display technique; and, (2) the model 
technique . 
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Tabular Display Technique. The tabular display tech- 
nique is used when the alternatives arc being evaluated by mea- 
sures of cost, or measures of effectiveness, or by both measures 
of cost and effectiveness that are incommensurate with one an- 
other. 

In this technique, the evaluation criteria underlying 
the analysis are identified at the tops of columns and arranged 
in decreasing importance of criteria from left to right as in- 
dicated in Figure 7. The alternatives are then listed verti- 
cally, with the alternative that meets the most significant cri- 
terion to the greatest extent listed first, and so on. This 
technique is useful when numerous alternatives are being eval- 
uated because the technique can be used to eliminate the less 
likely competitive alternatives and allow attention to be focused 
on the two or three major competitors. Whether the fixed-cost 
or fixed-effectiveness approach is used, the tabular array creat- 
ed will be very similar. In the fixed-cost approach, major em- 
phasis is placed on the effectiveness attainable; however, cost 
data indicating how the total cost is divided and phased is sig- 
nificant and should be shown. On the other hand, in the fixed- 
effectiveness approach, major emphasis is placed on ^ he total 
cost. The advantage of the tabular display technique is that 
the orderly presentation of alternatives cost and capability 
data is permitted so that their impact on the evaluation can 
be readily discerned and discussed along with the significant 
interrelationships. Conclusions, therefore, can be reached by 
visible traceable means. 



The Model Technique. The model technique, in which either 
cost models or effectiveness models are created, is usually used 
when the basic differences between the alternative systems are 
relatively minor, so as to permit the valid expression of their 
essential differences by a single parameter (Kazanowski, 1968, 
pp. 129, 134 5 136). These models may be structured along a 
spectrum of abstraction as shown in Figure 8. First, and the 
least abstract to be considered is the real-world system, with- 
in which the analyst cannot work. Next the more abstract models 
of this real-world arc examined. This includes the technique of 
operational exercise, gaming, simulation, and analytical model- 
ing. 'Last, at the opposite end of the spectrum, is total ab- 
straction, within which, again, the analyst cannot work. 

• Operational exercise technique. The closest 
approximation to the real -world would be an op- 
erational exercise, using the actual situations 
with the alternatives under study. Operational 
exercise differs from the real-world in that the 
scale is smaller and many of the personnel are 
simulated. In addition, the consequences of 
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failure are not considered. The operation exer- 
cise, consequently, is a very close approxima- 
tion to the real alternative but contains cer- 
tain factors that would not normally exist in 
the real world. The running cost of such an 
exercise is great and the number of alterna- 
tives which can be analyzed is small. 

• Gaming technique. The next region down this 
spectrum of abstraction is gaming. Here are 
removed from the representation of the real- 
world those components that can most easily be 
'simulated by a simple analog such as a compu- 
ter or some other device. 

• Simulation technique. The next stage towards 
model abstraction is to physically remove the 
human decision-maker from the representation. 
At this point the merits and operation of the 
alternatives are simulated on a computer in 
which the human decision rules are explicitly 
programmed, 

• Analytical technique. In those models already 
considered, although modified or compressed, 
there has been relationships between time in 
the real world and time in the model. In an- 
alytical models, even this relationship is 
abandoned. Here a system of equations is creat- 
ed: (1) that relate the characteristics of 
the alternative; and, (2) that are related to 
the measures of effectiveness. Examples of this 
are linear programming models, graphic models, 
and the whole class of equations referred to 

as math models . 

The model spectrum (Figure 8) represents, from top to bot 
tom, increasing abstraction from the real-world; or alternately, 
proceeding from bottom to top in the figure, increasing realism 
or closer approximation to the real world. It is also generally 
observed that both the cost of the analysis and the rate at 
which we can examine different situations decreases as the mo- 
del used becomes more abstract. For reasons of economy and 
completeness of analysis, wc would like to operate as close to 
total abstraction on this spectrum as possible* On the other 
hand, the possibilities of faulty assumptions increases as we 
move away from working in the real world. 

In exploring considerations that go into modeling tech- 
nique selection for a particular cost-effectiveness analysis it 
should be remembered that the basic function of the model is to 
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determine the value o£ the measure of effectiveness when the 
controllable variables of the alternatives are used (Blumstein, 
1969, pp. 34, 35 5 39). Here the term "model" is u.sed in a 
broad sense. Depending on the nature of the goals/objectives, 
the model used in the cost or effectiveness analysis may be for- 
mal or informal, very mathematical or not so mathematical, 
heavily computerized, moderately computerized, or not computer- 
ized at all. The main point, however, is that the model need 
not be highly formal and/or mathematical to be useful. The 
following important points should be considered in the selection 
of modeling techniques. 

• Remember that model building is an art, not a 
science. Often it is an experimental process. 

• Emphasize those factors relevant to the goals/ 
objectives, and suppress those factors that are 
relatively unimportant. The model is likely to 
be unmanageable if this is not done. 

• Develop a meaningful set of relationships among: 
(1) goals/objectives, (2) the alternatives 
available for attaining the goals/objectives, 
(3) the estimated cost of the alternatives, 
and (4) the estimated effectiveness for each 
of the alternatives. 

• Base the model design upon the "building-block" 
concept, which accepts analytical data prepared 
during prior steps in the creation of a set of 
smaller or partial models. Each small model 
will be used in the construction of a larger 

J ubsequent cost-effectiveness model (Packer, 
1968, p. 236) . 

• Make provisions for explicit treatment of un- 
certainty (see Step No. 10) (Fisher, 1967, p. 72), 

Within the spectrum of modeling technique (froxii concrete 
to highly abstract models), there are two techniques that are 
most appropriate for the evaluation of measures of costs and 
measures of effectiveness. It would seem that the choice should 
be dictated by the category of the cost-effectiveness study as 
well as the goals/objectives. This choice is between Simula* 
tion and analytical modclinj; techniques. Although the apparent 
difference between these two techniques is sometimes small, 
an analytical model is generally more efficient if the goals/ 
objectives can be stated in terms such that available mathema- 
tical or graphic techniques can be used to solve for an optimum 
solution (Packer, 1968, p. 235). Both analysis of cost and ef- 
fectiveness may make use of analytical models to represent: 
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(1) each alternatives being analyzed, (2) its operating 
characteristics, and (3) the management concepts of its op- 
eration and logistical support. These models are normally 
structured so that any parameter, or combination of parameters, 
can be varied to determine the relative effect on the alterna- 
tives* total cost and effectiveness (Heaton, 1969, p. 35). 



Model Considerations 

There are several considerations which should be consi- 
dered in the utilization of analytical models for both cost and 
effectiveness. 



Cost Analytical Model Considerations. The cost model 
relates the goals/objectives/requirements and activity rates of 
the alternative, and of the personnel that operate and/or sup- 
port it, to measures of cost (Bryk, 1965, p. 4). Mathematical 
cost models are the most frequently used to evaluate alterna- 
tives when goals/objectives can be mathematically formulated, 
when mathematical conditions can be met, and the solution can 
be computed (Miller § Rath, 1969, p. 18). The key advantage of 
the use of mathematical cost models is the speed with which a 
number of alternatives can be costed. IVhile exhibiting signi- 
ficant advantages for specific applications, mathematical cost 
models also possess substantial limitations. Two major dis- 
advantages are: 

• In creating the cost model a number of implicit 
assumptions are made. After the terms of the 
initial equation arc mathematically manipulated 
and condensed for efficient use, the initial 

Clements of the cost are no longer visible. i 
Thus, the analyst may forget his initial assump- 
tion and their limitations. 

) 

• Mathematical cost models arc relatively inflcxi- I 
ble and good basically for one concept and its 

minor variations. If a basic variation other 
than those accounted for by the model is to be 
costed, the model must be revised and possibly 
modified. This modification must be verified 
by comparison with conventional long^hand re- 
sults and as a consequence, the advantage of 
mathematical cost modeling is lost (Kazanowski, 
1968, p. 128). 

To conserve time, cost modeling is often perfomred concurrently 
with effectiveness modeling (Bryk, 196S, p. 14). 
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Effectiveness Analytical Model Considerations. The ef- 
fectiveness model relates: (1) the measures of eTfectiveness 
in achieving the goals/objectives to, (2) measures of the al- 
ternatives operational performance (Bryk, 1965, p. 2). The use 
of a mathematical effectiveness model is recommended when the 
alternatives under analysis are basically so similar that those 
evaluation criteria that cannot be readily validated, can be con 
sidered to cancel each other out. This leaves only the quanti- 
fiable and commensurable criteria for evaluation. It should be 
noted that this occurrence is relatively infrequent (Kazanowski, 
1968, p. 129). 

Analytical Model Computer Use Considerations. Analytic 
cost-effectiveness techniques have attained their present level 
of acceptance largely because of the availability of today's 
high-speed computers and the development of appropriate compu- 
ter models. The analytical models permit the study of complete 
interactions and many alternatives heretofore too complex and 
time-consuming for manual or desk-calculator analysis fUell, 
1964, p. 2). Prior to these computer programs, the cost-eTfec- 
tiveness analyst had to choose among several hand models, which 
were often unwieldy and unrealistic, and the older prototype 
computer models. These older computer models were frequently 
inflexible and failed to give insight into how conclusions were 
related to the assumptions (Quade, 1968, p. 243). The computer- 
ization of current analytical models is warranted, however, only 
when the analysis of alternatives involves repeated computation 
of complex functions (or when the same sub-routines are performe 
repeatedly). In addition, computerized analytical models of ad- 
vanced future alternatives are oftentimes of limited value be- 
cause the expensive programming is time-consuming and soon ob- 
solete by today's standards. 

Concerning cost analytical modeling, if the model is sim- 
))le, it does not need to be computerized: if it is complex, 
the maintenance of the computer model and associated documenta- 
tion may become time-consuming and expensive. A possible com- 
promise might be to computerize those portions of an analytical 
cost model that are not subject to frequent revision. In a 
study directly related to instructional cost modeling, it was 
indicated that complex mathematical modeling would be incompa- 
tible with the concept of instructional cost-effectiveness devel 
opment. This concept was to develop a procedure that was easy 
to use and not voluminous (Institute for Educational Development 
1970, pp. 3-4). Regarding effectiveness analytical models, thes 
are seldom mathematical and as a consequence, computerization is 
seldom applicable. 



Analytical Model Handbook Considerations. Rather than 
computerize generalized all-purpose analytical models, it has 
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been found more functional to compile an analytical costing and 
effectiveness handbook . Graphical displays of cost and effec- 
tiveness relationships , along with their underlying goals/objec- 
tives/requirements and assumptions, can be readily compiled and 
utilized. The handbook both allows and makes publicly visible 
the exercise of judgment to ascribe the impact of subtle devia- 
tions from a basic or common design on the cost and effective- 
ness of various alternatives. The costing/effectiveness hand- 
book also has the virtue of simplicity, speed, flexibility, and 
economy (Kazanowski, 1968, pp. 131-132). The Institute for Ed- 
ucational Development has developed such a handbook, for alter- 
native instructional design selection, under contract for the 
Bureau of Naval Personnel and naval training schools (Institute 
for Educational Development, 1970, p. 1). This two-part docu- 
ment is noteworthy in both volume and level of detail. Another 
approach in the development of an analytical model handbook is 
a document distributed by the Department of Defense (Department 
of Defense, 1969, p. 1). This both outlines the procedure of 
analysis and encloses analytical forms to be used. 

It should be emphasized the cost-effectiveness models can 
never be completely realistic for they are dependent on too many 
uncertain parameters. The reliance on expert judgment is indis- 
pensable to all analysis. Moreover, the virtue of cost-effec- 
tiveness analysis is the provision of a framework that allows 
the judgment and intuition of specialists in diverse fields to 
be combined. This framework is the analytical model which re- 
presents a simplified, stylized representation of those aspects 
of the real world as appropriate to the goals/objectives (Quade, 
1968, pp. 243, 246-247) . 



Step 9 

Analyze merits of alternatives. In this final step, the 
integration of the previously derived alternative cost and effec 
tiveness analytical models is performed* The purpose of the in- 
tegration is to combine the expected values of alternative cost 
and effectiveness into a single common framework (Seller, 1969, 
p. 71). This framework may be based on either the tabular dis- 
play technique or model technique discussed in Step No. 8. 

An example of the tabular display technique is found in 
Figure 9. This figure is, in addition, an example of the appli- 
cation process of cost effectiveness analysis (previously dis- 
cussed) in which the analysis is both structured and evaluation 
is completed. This display was developed consistent with infor- 
mation provided by a cost and effectiveness input similar to 
that found in Figure 7. The instructional designs under study 
represents a variety of alternatives for in-service employee 
training. These alternatives are listed in order of recommended 
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preference according to the cost-effectiveness ^selection cri- 
teria (Drouet, 1968, p. 221) on an evaluative scale of 1 to 6 
(6 equals the highest or most preferred) . 

An example of the model technique is presented in Figure 
10. This example illustrates both a graphic analytical model 
and the procedural process (also previously discussed) of cost 
effectiveness analysis. In this case, the analytical model il- 
lustrates the logic and procedure of the analysis of each alter- 
native instruction design for service training. The model in- 
dicates that each alternative instructional design may be char- 
acterized by: (1) instructional requirements , (2) instruction 
time, (3) course duration, (4) student achievement, (5) facil- 
ity requirements, (6) number of courses per year, (7) equip- 
ment, (8) course development, (9) instructional materials, (10) 
software development, and (11) student and instructor travel 
time. In addition, each instructional design can be distinguished 
by: (1) student achievement, (2) student attrition rate, (3) 
student proficiency on the job after graduation, and (4) student 
morale. Attributes concerning student performance and morale, 
during instruction or later, are indentified as measures of ef- 
fectiveness. This terminology is used because the effectiveness 
of the instructional design is measured in terms of the end pro- 
ducts; i.e., student course completion. Other factors, such as 
instructor time per course and amount of instructional materials 
required are classified as measures of efficiency. The analy- 
tical model indicates that feasibility of the instructional de- 
sign alternatives are determined initially. Next, the character- 
istics of the feasible alternatives are listed as well as the 
differences in the alternatives' effectiveness, efficiency, and 
costs. Risk factors are also considered in the analytical model. 
Last, the criterion for selecting the instructional design is 
applied (Institute for Educational Development, 1969, pp. 23, 
25 ^ 26). This criterion is as follows: 

If the sum of incremental benefits in dollars of the new 
program exceeds its incremental benefits in dollars of 
the ntfw program (sic., instructional disign) . . . exceeds 
its increase in incremental costs (with acceptable risk) 
then the new program is ranked higher than existing pro- 
gram and the existing program is eliminated from further 
consideration. If the reverse is true, the existing pro- 
gram is rated higher and the new program is eliminated 
(Institute for Educational Development, 1969, p. 29). 

Regardless of the technique used, both the cost-effective- 
ness tabular display technique and the analytical model technique 
relate costs and effectiveness so that the merits of the alterna- 
tives may be analyzed for the selection of the preferred alterna- 
tive (Bryk, 1965, p. 4). 
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Once the alternatives are arranged in order of their 
acceptability in relationship to the evaluation criteria, it 
is generally possible to eliminate the most obviously poorest 
competitors and focus attention on the top three or four. If 
the effectiveness and cost evaluation outcomes for the top al- 
ternative are consistently superior to the respective values 
of the other alternatives, then that alternative is the pre- 
ferred selection. If the performance values for the top alter- 
natives are virtually identical, and no significant difference 
in cost exists, the appropriate conclusion may be that there is 
no significant difference between the top candidates. In this 
case, the adoption of parallel study or development efforts of 
both top candidates may be indicated in order to identify the 
preferred alternative. This would mean that one alternative 
would be selected at a later date. If the costs of the compet- 
ing alternatives differ significantly, and the evaluation rat- 
ings also vary significantly, the selection may need to be made 
on the basis of personal value judgments (Kazanowski, 1968, p. 
135) . 



Step 10 

Analyze uncertainty factors. All cost-effectiveness an- 
alysis studies include doubtful features that may not be satis- 
factorily specified or quantitatively resolved within the study 
itself (Brechner, 1967, p. 57). Important decision problems 
involve major elements of uncertainty. Consequently, a cost- 
effectiveness analysis of such problems must provide for the ex- 
plicit treatment of uncertainty for consideration by the decision- 
maker. 

Two main types of uncertainty may be distinguished: (1) 
uncertainty about the state of the world in the future, and (2) 
statistical uncertainty about the data on the present state of 
the alternative. State-of -the-future uncertainty stems from 
chance elements in the real world and would exist even if there 
were no ties of the state to the world of the future (Fisher, 
1967, pp. 72-73). Statistical data uncertainty is most closely 
associated with cost -effectiveness analysis because, unfortun- 
ately, the choices between alternative candidates are seldom 
made on the basis of clear-cut data. Factors which add to the 
uncertainties include: (1) alternatives are frequently inade- 
quate to fully attain the objectives, (2) measures of effective- 
ness may not actually measure the extent to which objectives are 
attained, (3) predictions from the cost-effectiveness model are 
apt to be full of uncertainties, and (4) other criteria which 
look almost as attractive as the criteria chosen may lead to a 
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different order of preference (Quade, 1969, p. 5). Three types 
of uncertainty analysis are most often used to treat statisti- 
cal uncertainties. These are: (1) sensitivity analysis, (2) 
contingency analysis, and (3) a Fortiori analysis. 



Sensitivity Analysis. In many instances the output of 
a cost-effectiveness analysis is very sensitive to the assump- 
tions made. The conclusions reached may be unknowingly yet 
significantly biased by some apparently innocuous assumptions 
which are essential to the analysis (Kazanowski, 1968, p. 138). 
For example, suppose in a given analysis there are several un- 
certain key parameters. Instead of using "expected values" for 
these parameters, the analyst could test for the influence of 
these assumptions by substituting several values (i.e., high 
and low) in an attempt to see how sensitive the results (the 
ranking of the alternatives being considered) are to variations 
in these uncertain parameters (Fisher, 1967, p. 73). Another 
example would be an instance in which the alternatives are near 
ly equal in merit. Here, it would be desirable to attempt to 
estimate the cost-effectiveness that would be derived from the 
candidate alternatives if they were cancelled at three or four 
different major life-cycle schedule milestones. It is also de- 
sirable to perform this analysis even when one candidate alter- 
native appears to be clearly superior with respect to both cost 
and effectiveness criteria. Such an analysis may point out the 
need for caution in making an otherwise unqualified endorsement 
of the alternative (Kazanowski, X968, pp. 138-139). 



Contingency Analysis. This type of analysis investigates 
how the ranking of the candidate alternatives maintains ranking 
when a relevant change in criteria for evaluating the alterna- 
tives is postulated, or a major change in the general environ- 
ment is introduced. These techniques assess the degree to which 
results are contingent upon any one factor, or each of several 
factors . 



A Fortiori Analysis. This type of analysis would be ap- 
plicable when in a planning decision problem the generally ac- 
cepted intuitive judgment strongly favors alternative X. The 
analyst feels, however, that X might be a poor choice and that 
alternative Y might actually be preferable. In performing an 
analysis of X versus Y, the analyst may choose deliberately to 
resolve the major uncertainties in favor of X and see how Y 
compares under these adverse conditions. 



47 



Although the above three types of uncertainty analysis 
may be useful in a direct sense, they may also contribute in- 
directly. For example, through sensitivity and contingency an- 
alysis the analyst may improve his understanding of critical 
uncertainties in a problem area. On the basis of this knowledge 
he might then be able to create a new alternative that would 
assume more protection against a wider range of uncertainties. 
This is often difficult to do but when it can be accomplished, 
it offers one of the best ways to compensate for uncertainty 
(Fisher, 1967, pp. 73-74). 



Step 11 

Document bases of previous steps and submit findings. 
The product of a cost-effectiveness analysis study, will in all 
probability be a report to the decision-maker. In support of 
this report a key element of systematic cost-effectiveness an- 
alysis is sufficient documentation of methods, assumptions, 
sources, etc., so that another analyst would achieve substan- 
tially the same results with the same material. Without such 
documented results, a cost-effectiveness analysis appeal for 
acceptance rests solely on faith in the authority and expertise 
of the analyst without critical examination of the way in which 
he arrived at the recommendations (Heymont, 1965, p. 20). 

Particular emphasis should be placed on the adequacy of 
documenting the following: 

• Specific goals/objectives to be attained. 

• Essential requirements of those goals/objectives 
along with associated assumptions. 

• Alternative capabilities and associated assump- 
tions . 

• Alternative costs and associated assumptions 
(learning curves, time, quantities, etc.). 

• Alternative evaluation and associated assumptions 
(scenarios , criteria, etc . ) . 

• Conclusions: The recommended alternative, the 
limitations, and the associated uncertainty fac- 
tors . 

The use of highly esoteric mathematics for documentation 
should be discouraged. With effort, imagination, and forethought, 
the analyst can usually suitably portray complex mathematical and 
functional relationships in simplified and perhaps graphic form. 
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No judicious decision-maker can be expected to endorse a con- 
clusion or recommendation whose rationale and derivation he can 
not fully understand. It is the responsibility of the analyst 
to present the documentation in an appropriate and understand- 
able manner. To have a high probability of acceptance by the 
decision-maker all elements of the cost -eff ectiveness analysis 
must be documentated in such a manner that the entire process 
can be clearly followed (Kazanowski, 1968, pp. 139-140). 
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COST-EFFECTIVENESS ANALYSIS 
DIFFERENCES AND CHARACTERISTICS OF METHODS 



From the instructional cost -effectiveness analyst's 
point of view, the selection of a study category and an analy- 
tical process to be used is determined by requirements outside 
of the control of the analyst. These requirements are deter- 
mined either by the requestor of the study or by the type of 
study itself. Only within the procedure of the cost-effective- 
ness analysis itself does the analyst have a choice in analyti- 
cal technique. This standard procedure for cost-effectiveness 
analysis underlies the methods previously described. These meth- 
ods contained certain functional elements with differing charac- 
teristics which should be considered in any standard step proce- 
dure used by the analysis. Latitude is acceptable in the manner 
and procedure of analytical steps for a certain amount of over- 
lap and duplication are present in the steps. Figure 11 illus- 
trates the function, differences, and characteristics of the 
various cost-effectiveness methods. 



FUNCTION 

The following functional elements are considered necessary 
in the performance of a cost -eff ectiveness analysis. 

• Step 1. Review definition of general goals/ob- * 
ject ives. 

• Step 2. Review identification of parameters or 
requirements . 

• Step 3, Review development of alternatives. 

• Step 4. Hstablish cost and effectiveness eval- 
uation criteria.. 

• Step 5. Determine cost of alternatives. 

• Step 6. Determine capabilities or effectiveness 
of alternatives* 

• Step 7. Select fixed^cost or fixed-effectiveness 
approach, 

• Step 8. Create alternatives versus criterion array* 
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COST-EFFECTIVENESS ANALYSIS STANDARD PROCEDURE 



FUNCTION 


nTcccppHrpc 
UlrrtKt.Ml*to 




STEP 1 

REVIEW DEFINITION 
OF GENERAL GOALS/ 
OBJECTIVES 






STEP 2 

REVIEW IDENTIFICA- 
TION OF PARA- 
METERS OR REQUIRE- 
MENTS 






STEP 3 

REVIEW DEVELOPMENT 
OF ALTERNATIVES 






STEP 4 

ESTABLISH COST 
AND EFFECTIVENESS 
F VAI IJAT T OH 

CRITERIA 


COST MEASURES 

• NON-TIME 

pu AC rn 
r n n O b U 

ANALYSIS 
t TIME-PHASED 
ANALYSIS 


CHOICE DETERMINED 
BY CATEGORY OF 
STUDY 




EFFECTIVENESS 
MEASURES 

• PERFORMANCE 
IDENTIFIERS 

• STANDARDS 

• UNIQUE 
EDUCATIONAL 


CHOICE DETERMINED 
BY GOALS/OBJEC- 
TIVES/REQUIREMENTS 


STEP 5 

DETERMINE COST 
OF ALTERNATIVES 







Note: After Pearson, 1972, p. 112 



Figure 11. Cost-Effectiveness Analysis Functions, Differences, 
and Characteristics (Sheet 1 of 3) 
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FUNCTION 


DIFFERENCES 


CHARACTERISTICS 


STEP 6 

DETERMINE CAPABILI- 
TIES OR EFFECTIVE- 
NESS OF ALTERNA- 
TIVES 






STEP 7 

SELECT FIXED-COST 
OR FIXED-EFFEC- \ 
TIVENESS APPROACH 






STEP 8 

CREATE ALTERNA- 
TIVES VERSUS 
CRITERION ARRAY 


TABULAR DISPLAY 
TECHNIQUE 


CHOICE DETERMINED 
BY EVALUATION 
CRITERIA* 
RECOMMENDED WHEN 
CRITERIA VARY 
WIDELY 




MODEL TECHNIQUE 

• OPERATIONAL 
EXERCISE 

• GAMING 

• SIMULATION 


NOT RECOMMENDED 
FOR INSTRUCTIONAL 
COST-EFFECTIVENESS 
ANALYSIS BECAUSE 
OF COMPLEXITY 




• ANALYTICAL 


RECOMMENDED FOR 
USE WHEN THE GOALS/ 
OBJECTIVES/REQUIRE- 
MENTS CAN BE 
STATED IN TERMS 
SUCH THAT MATHEMA- 
TICAL AND GRAPHIC 
TECHNIQUES CAN BE 
USED 
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Figure 11. Cost-Effectiveness Analysis Functions, Differences, 
and Characteristics (Sheet 2 of 3) 



FUNCTION 


DIFFERENCES 


CHARACTERISTICS 


STEP 9 

ANALYZE MERITS 
OF ALTERNATIVES 


TARIII AD ni^PI AY 

TECHNIQUE 


CHOICE DETERMINED 
BY EVALUATION CRI- 
TERIA; RECOMMENDED 
WHEN CRITERIA VARY 
WIDELY 




MODEL TECHNIQUE 

• OPERATIONAL 
EXERCISE 

• GAMING 

• SIMULATION 


NOT RECOMMENDED FOR 
INSTRUCTIONAL 
COST-EFFECTIVENESS 
ANALYSIS BECAUSE 
Of COMPLEXITY 




• ANALYTICAL 


RECOMMENDED FOR USE 
WHEN THE GOALS/ 
OBJECTIVES/REQUIRE- 
MENTS CAN BE 
STATED IN TERMS 
SUCH THAT MATHE- 
MATICAL AND GRAPHIC 
TECHNIQUES CAN BE 
USED 


STEP 10 

ANALYZE UNCERTAIN- 
TY FACTORS 


SENSITIVITY 
ANALYSIS 


RECOMMENDED FOR USE 
WHEN SELECTION 
CONCLUSION MAY BE 
BASED ON ASSUMPTIONS 




CONTINGENCY 
ANALYSIS 


RECOMMENDED FOR USE 
WHEN EVALUATION 
CRITERIA MAY BE 
UNCERTAIN 




FORTIORI 
ANALYSIS 


RECOMMENDED FOR USE 
WHEN SELECTION MAY 
BE BASED ON INTUI- 
TIVE JUDGMENT 


STEP 11 

DOCUMENT BASIS OF 
PREVIOUS STEPS 
AND SUBfUT 
FINDINGS 







Figure 11. Cost-Effectiveness Analysis Functions, Differences, 
and Characteristics (Sheet 3 of 3) 
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• Step 9. Analyze merits of alternatives. 

• Step 10. Analyze uncertainty factors. 

• Step 11. Document basis of previous steps and 
submit findings. 



DIFFERENCE 

Differences occur in the performance of various cost-ef- 
fectiveness analysis in several functional elements of the stan- 
dard procedural steps. These differences are found and described 
in some detail in Step 4, Step 8, Step 9, and Step 10 of Figure 
11. 



CHARACTERISTIC 

Characteristics of the differences in Step 4, Step 8, 
Step 9, and Step 10 of the functional elements of the standard 
procedure are also considered in Figure 11. It is noted, how- 
ever, that the principle differences in cost-effectiveness analy- 
sis methods are in Step 8 (create alternative versus criterion 
array) and in Step 9 (analyze merits of alternatives). These 
two differences are actually the same; i.e., the choice of using 
the tabular display technique or the analytical model technique. 
This is the only methodological choice available to the cost-ef- 
fectiveness analyst not determined by others, determined by the 
type of study itself, or influenced by outside conditions. 



COST-F FECTIVENESS ANALYSIS 
SELECTIU., OF TECHNIQUES FACTORS 



Only within certain functional elements of the standard 
step procedure is there a choice of analytical techniques that 
may be used. In general these are choices between the tabular 
display technique and the analytical model technique. The tab- 
ular display technique may be used for all applications, but 
the analytical model technique is recommended for use only when 
the alternative criteria do not vary greatly and when the goals/ 
objectives/and requirements can be stated in terms that will 
adapt themselves to mathematical and graphic techniques. 

Other advantages and disadvantages in the selection of 
either the tabular display technique or the analytical model 
technique should be considered in the selection of one of these 
techniques for instructional cost-effectiveness analysis. Fig- 
ure 12 exhibits these considerations, which are as follows: 



TABULAR DISPLAY TECHNIQUE 

The advantages of this technique are: (1) ease of eval- 
uation, (2) traceable data conditions, (3) low cost, (4) may 
be used for all categories of analysis as well as both cost and 
effectiveness evaluations, and (5) easily modified to meet new 
requirements . 

The disadvantages of the tabular display are: (1) un- 
realistic to the real world, (2) does not have computer capa- 
bility, if required, and (3) an easily understood graphic form 
must be created for presentation to the decision-maker. 



ANALYTICAL MODEL TECHNIQUE 

The advantages of this technique are: (1) computer capa- 
bility, if required, but need not be computerized, (2) lowest 
cost for models and simulation, (3) speed in cost modeling and 
evaluation, and (4) as a by-product of the analytical model 
process, may have an easily understood graphic form for presen- 
tation to the decision -maker. 

The disadvantages of the analytical model technique are: 
(1) data traceability is lost in the analytical process, (2) 
limited to analysis between similar alternatives, (3) not 
appropriate for effectiveness evaluation, and (4) inflexible 
to change. 
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CHOICE OF TECHNIQUE CONSIDERATIONS 



TECHNIQUE 


ADVANTAGE 


DISADVANTAGE 


TABULAR 
DISPLAY 
TECHNIQUE 


A. EASE OF 
EVALUATION 

B. DATA CONDITIONS 
TRACEABLE 

C. LOW COST 

D. USED FOR 

ALL CATEGORIES 
OF ANALYSIS 
AS WELL AS 
BOTH COST AND 
EFFECTIVENESS 
EVALUATIONS 

E. EASILY MODIFIED 
TO MEET NEW 
REQUIREMENTS 


A. UNREALISTIC 

B. COMPUTER 
CAPABILITY NOT 
AVAILABLE 

C. EASILY UNDERSTOOD 
GRAPHIC FORM FOR 
DECISION-MAKER 
MUST BE CREATED 


ANALYTICAL 

MODEL 

TECHNIQUE 

• 


A. COMPUTER CAP- 
ABILITY, BUT 
NEED NOT BE 
COMPUTERIZED 

B. LOWEST COST FOR 
MODELS 

C. SPEED IN COST 
MODELING AND 
EVALUATION 

D. EASILY UNDERSTOOD 
•GRAPHIC FORM FOR 

DECISION-MAKER 
AVAILABLE 


A. LOSS OF DATA 
TRACEABILITY 

B. LIMITED TO ANAL- 
YSIS BETWEEN 

C T U T 1 AD 

ALTERNATIVES 

C. NOT APPROPRIATE 
FOR EFFECTIVENESS 
EVALUATION 

D. INFLEXIBLE TO 
CHANGE 



Note: After Pearson, 1972, p. 141 



Figure 12. Cost-Effectiveness Analysis Selection of Techniques 
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COST-EFFECTIVENESS 
APPLICATION TO INSTRUCTIONAL DEVELOPMENT 



This monograph has provided for the instructional design- 
er procedural steps to follow in the performance of instruction- 
al cost-effectiveness analysis. In addition, the procedural ele- 
ments may be used as a checklist in the evaluation of existing 
instructional cost -effectiveness studies performed by others. 



IMPLICATIONS 

The direct implication of cost-effectiveness as a decision- 
assisting tool for instructional development arc that: (1) the 
performance of cost-effectiveness analysis may be formal or in- 
formal, (2) the analysis time involved may be 30 minutes or a 
week, (3) all of the procedural elements may or may not be con- 
sidered; and (4) the degree of detail explored may or may not 
be great. Instructional cost-effectiveness analysis is best per- 
formed as a service by someone other than the final decision-maker. 
This person may be an instructional designer or as a highly spe- 
cialized service to the designer. The final decision-maker served 
most often might be: (1) an academic dean or department head, 
(2) a training manager or supervisor in industry or government, 
or (3) a public school system superintendent or instructional 
program director. At the present state of development and accep- 
tance, cost-effectiveness analysis seems most applicable to in- 
structional development in what might be called "adult or continu- 
ing education." The compensatory, vocational, and business in- 
structional programs in school systems could benefit most immedi- 
ately from a systems/cost analysis. It is further thought that 
merit pay, performance contracting, and accountability in instruc- 
tion could provide the impetus to bring cost-effectiveness into 
public school systems. 



PROBLEM AREAS 

It seems worthwhile to note certain general areas of unique 
problems associated with the use of cost-effectiveness analysis 
in the selection of alternatives during instructional design and 
development. These are areas in which problems may be overcome 
by further experience and information-sharing. 

• Software (Manual, semiautomatic, and automatic) 
must be developed which will allow updating of 
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the cost and effectiveness model consistent 
with experience, and improvement in the cost- 
effectiveness model itself. 

• More experience is needed in the design of 
cost-effectiveness modeling as well as cost- 
effectiveness user training at the local ap- 
plication level (Filep, 1970). 

• Current data is inadequate for both cost and 
effectiveness modeling due to the lack of his- 
torical records (Institute for Educational 
Development, 1969, p. 49). Records on either 
budget or performance estimates versus com- 
pliance are not being maintained. Perhaps 
wliat is needed is the desi;;n of some report- 
ing instrument in a format for instructional 
cost and effectiveness data extraction (Filep, 
1970) . 

• Additional research is needed before cost-ef- 
fectiveness analysis can be used with under- 
standing. 



RECOMMENDATIONS 

The following recommendations are offered to assist fur- 
ther research) efforts of others in the area of instructional 
cost-effectiveness analysis . 

Changes in cost-effectiveness nomenclature and terminology 
that have the same prior meaning should be noted. Government ac- 
tivities represent the largest concentration of potential clients 
for operations research, systems/cost analysis, and the like. 
Because of changes in political administration, these activities \ 
will be in frequent flux as to what today* s terminology is "in." 
As a consequence, today* s trend seems to be away from the term- 
inology so popular during the 1960*s. It is suggested that per- 
haps a more meaningful terminology could be designated for in- 
\ structional design purposes. ^ 

Instructional cost-effectiveness analysis has not received 
either the credit or publicity earned. No reports from public 
school systems using cost-effectiveness analysis were found. If 
cost-effectiveness analysis is indeed used to the extent reported 
in the study (48 percent), it would seem that more should be pub- 
lished on the performance and result of the cost-effectiveness 
method itself. It is suggested that more instructional designers 
with cost-effectiveness expertise disseminate their experiences. 
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More information is needed on measures of performance 
regarding both cost and effectiveness. Measures of the con- 
sequences of failure (lack of effectiveness) should be also 
considered. It would be desirable if unique measures of per- 
formance could be developed for instructional design that were 
general enough for differing applications. It is recommended 
that some sort of "clearing house" or responsible organization 
acquire, maintain, and disseminate historical performance mea- 
surement information on instructional cost and effectiveness. 

An analytical model for instructional cost-effectiveness 
that could be easily modified for differing applications is 
needed. This model should be designed with computer capability 
and available operational instructions. It is recommended that 
such a computerized analytical model be developed, and be made 
available to clients on a commercial basis. 

Research on the direct application of instructional cost- 
effectiveness is almost nonexistent. It is suggested that sev- 
eral aspe'^ts of this study are applicable for further in-depth 
study, with an emphasis on real-world application. 
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SUMMARY 



Much has been reported on the advantages of cost-effec- 
tiveness analysis as a decision-assisting tool for instructional 
development. Little, however, has been reported regarding the 
actual performance of the analysis itself. Although a variety 
of cost-effectiveness analysis methods exists, it was theorized 
that one method might be more applicable to the unique require- 
ments of instructional design and development. The purpose of 
the monograph, therefore, is to identify and analyze the various 
cost-effectiveness analysis methods so that the most appropriate 
method could be selected as a decision-assisting tool. 

What appeared to be a large variety of cos t-ef f cctivcncss 
analysis methods is due to choices (1) in the "level" of analy- 
sis, (2) in the process of analysis, and (3) in the analytical 
methods employed. The choices of "level" and process of analy- 
sis are usually outside of the control of the cost-effectiveness 
analyst. This is because both "level" and process of analysis 
are determined by the requestor or by the nature of the study 
itself. Within the analytical methods, however, the procedures 
used in the cost-effectiveness studies are described in terms 
of 11 procedural steps. Within these procedural steps only a 
limited number of pathways or alternative techniques are options 
available to the analyst. These options were determined by the 
characteristics of two analytical techniques approaches. These 
two approaches were the tabular display technique and the model 
technique. The tabular display technique was found recommended 
for all cost-effectiveness applications but the model technique 
was limited for use when the alternative criteria did not vary 
greatly, and when the goals/ob jectives/rcquiremcnts could be 
stated in terms that would adapt to mathematical and graphic 
techniques. In addition, the tabular display technique is re- 
commended as a dcci sion-assistin)» tool for instructional design 
and development because of the characteristics of easily under- 
stood results and traceable data conditions. 

In the selection of alternative instructional design, the 
following major considerations are involved in performing cost- 
effectiveness analysis : 

# Proper structuring of the problem. The analysis 
addresses itself to the goals/objectives defini- 
tion as well as requirement identification. 
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• An appropriate procedure and systematic analy- 
tical framework must be employed in making com- 
parisons. Measures of cost and effectiveness 
must be identified and criteria established so 
that subsequent and capabilities data may be 
appropriately modeled. 

• Model construction (either formal or informal) 
is usually necessary in the analytical process. 
The main purpose of the model is to develop 
rclntlonships among objectives, the relevant 
alternatives available for attaining the ob- 
jectives, tlie estimated cost of tlic alterna- 
tives, and tlie estimated performance capabili- 
ties for each alternative. 

• Because the model is only an abstract represen- 
tation of reality, it is desirable to perform 
validity checking of the analytical procedure; 
e.g., how well can the model describe known 
facts and situations? 

• Uncertainty factors regarding the analysis must 
be indentified . 

Once accomplished the instructional cost-ef f ectiveness analysis 
and the resulting alternative recommendation can be a signifi- 
cant addition to decision-assisting tools. 
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APPENDIX 



DEFINITION OF TERMS USED 

Cost. Unit of resource that is the limiting constraint. 
Dollars are used in most cases but other resources, such as 
manpower, materials, and facilities will also be considered as 
measures of cost (Ilatry, 1969, p. 44). 

Cost-benefit analysis. A problem solving approach which 
requires the definition of objectives, and identification of 
the alternative that: (1) yields the greatest benefit for a 
given cost; or what amounts to the same thing, (?) that yields 
a required amount of benefits for the least cost. The term 
usually applies to situations in which the alternative outputs 
can be quantified in dollars . A chief characteristic is that 
the aim is to calculate the present value of benefits and costs, 
subject to specified constraints. 

Cost-effectiveness analysla. A problem solving approach 
which requires the definition of objectives, identification of 
alternative ways of achieving the objective, and identification 
of the alternative that: (1) yields the greatest effectiveness 
for any given costi or what amounts to the same thing, (2) that 
yields a required degree of effectiveness for the least cost. 
The term is usually used in applications in which the alterna- 
tive outputs cannot be easily quantified in dollars (McGivney 
and Nelson, 1969, p. ICS). 

Cost-utility analysis. Same as co ^t-ef fcctiveness analy- 
sis (Anshen, 1967, p. 3). Numerous terms currently convey the 
same general meaning (i.e., systems analysis, operations analy- 
sis, cost-effectiveness analysis, and operations research); how- 
ever, they have varying connotations to different people (Fisher, 
1967, p. 66). Because of such confusion in terminology and 
meaning, cost-effectiveness will be the term used to reflect the 
level of intent of the study. ^ 

Discounted cost. An analytical technique that accounts 
for the fact that money to be paid in the future yields invest- 
ment return until the point in time when it is actually spent; 
consequently, present money spent is worth less than future mon- 
ey spent (Seiler^ 1969, p. 17). 

Effective. The accomplishment of the recognized objec- 
tives (Beynon, 1968, p. 84). 
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Initial cost. Investment in goods and services required 
to establish and operate a system. These goods have a useful 
value of longer than a year; thus the costs are not repeated 
every year (Esscff, 1970), 

Life cycle costs. Combined initial costs and operational 
costs for the estimated useful life of the system (Esseff, 1970). 

Model* Relations used to portray real or expected con- 
ditions, actions, or effects in order to predict the outcome of 
actions (Heymont, 1965, p. 56). 

Objective, A statement that descri1)cs in observable and 
measurable terms the expected output performance of the product 
of the system (Banathy, 1968, p. 89). 

Planning , programming, and budget i ng system (PPBS). Cost- 
benefit analysis on a large scale (Commission on Instructional 
Technology, 1970, p. 90), A conceptual approach to decision-mak- 
ing which emphasizes outputs, program activities, and accomplish- 
ments. Long-range planning, analytic evaluative tools, and eco- 
nomic rationality are basic ingredients (Hartley, 1968, p. 258). 

System. An integrated relationship of people, equipment, 
and methods appropriately organized to accomplish defined tasks 
(Hcymont , 1965, p. 59) . 

Systems analysis (systems approach). Self -correcting and 
logical methodology of decision-making to be used for the design 
development of manmade entities. Strategies of this methodology 
include: (1) the formulation of performance objective, (2) the 
analysis of functions and components, and (3) implementation 
(Bannthy, 1968, p. 91) . 

Time-phased cost* The presentation of costs by the time 
period in which thr costs occur rather than a total cost figure 
(lleymont, 1965, p, 60) . 

TraJe-off . The weighing of alternative means to be em- 
ployed for the accomplishment of required functions (Banathy, 
1968, p. 91). Cost-effectiveness analysis represents one aspect 
of trade-off analysis • 
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